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Abstract 

The two purposes of the pre-post naturalistic research design were to: 1) Investigate the impact of 
positive behavioral interventions and supports (PBIS) on the behavioral functioning of students with 
emotional disturbance (ED) (TV = 37) served in self-contained settings; and 2) examine the extent to which 
teacher fidelity of PBIS implementation influenced student changes in behavioral functioning over the 
course of a school year. Results revealed significant reductions in externalizing and total problem 
behaviors for the students. Additionally, teacher fidelity to PBIS played a large and statistically 
significant role in improving the behavior of students with emotional disturbance. Limitations of the 
design and implications of the findings are discussed. 

Keywords: Emotional disturbance, positive behavioral supports, treatment fidelity, teacher professional 
development 


Introduction 

The task of addressing emotional and behavioral problems within school settings is complex. In 
the general school population, it is estimated approximately 20% of children and adolescents have serious 
emotional and behavioral problems (Hoagwood & Erwin, 1997; Department of Health and Human 
Services, 1999). These students present behaviors that range in both severity and kind, requiring a broad 
continuum of responses by school personnel in order to adequately address their behavioral needs. Many 
schools have adopted a three-tiered model of positive behavioral interventions and supports (PBIS) as a 
manner in which to decrease the occurrence of a range of challenging behaviors and to promote positive 
learning outcomes (Walker et al., 1996). The PBIS approach includes three tiers. Tier 1 implements 
primary prevention efforts that address school and classroom-wide universal interventions to support 
positive behavior for all students, Tier 2 implements secondary targeted interventions for use with 
students with at-risk behaviors, and Tier 3 implements tertiary or intensive individualized interventions 
designed for students with the most severe behavioral issues (Sugai & Homer, 2002). 

School-wide PBIS is an evidence-based prevention model used to reduce challenging behavior 
and promote safe and healthy learning environments in schools (Sugai & Homer, 2002; McCurdy, 
Mannella, & Eldridge, 2003). Results of an analysis of the barriers and facilitators to the successful 
implementation of school-wide PBIS indicate that staff buy-in is one of the primary barriers for school- 
level implementation (Kincaid, Childs, Blase, & Wallace, 2007). The need for effective staff training has 
also been highlighted as a potential barrier or facilitator to successful implementation of PBIS. Staff buy- 
in and staff training are necessary but not sufficient indicators of successful PBIS implementation. 

Perhaps the most important variable to achieving positive and sustainable outcomes associated with PBIS 
is building the capacity of educators to implement evidence-based interventions within the PBIS model 
with fidelity. Sugai and Homer (2008), for example, refer to the importance of fidelity when 
implementing PBIS, and state that “schools are able to implement SWPBS [PBIS] with fidelity when 
supported by local trainers and coaches” (p. 71). Fidelity of implementation is traditionally defined as the 
extent to which the intervention is implemented as designed during an experimental study (e.g., Hord, 
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Rutherford, Huling-Austin, & Hall, 1987). More specifically, fidelity of implementation can be 
differentiated into two primary categories: a) fidelity of structure (i.e., adherence and exposure), and b) 
fidelity of process (i.e., program differentiation, quality of delivery and responsiveness) (Dane & 
Schneider, 1998; Mowbray, Holter, Teague, & Bybee, 2003). Although widely reported in the health 
literature, the influence of fidelity of implementation is rarely reported in studies of K-12 core and 
intervention programs (O’Donnell, 2008). 

For the majority of general educators, fidelity of the process of PBIS implementation will be most 
critical at the primary prevention level of intervention, including school-wide, classroom, and non¬ 
classroom settings, as the majority of students encountered will not be at risk or have an elevated risk of 
antisocial behavior (Walker, et al., 1996). In contrast, teachers of students with the most challenging 
behaviors (e.g., emotional disturbance), or displaying signs of antisocial behavior patterns, need to carry 
out the process of PBIS with fidelity comprehensively, or at all three levels of prevention. These levels of 
prevention include the structure and process of functional behavioral assessments, data-based decision 
making, individualized response to intervention systems, de-escalation of behavior events, linking 
community supports to fa mi lies and youth needs, self-monitoring, and behavioral measurement (e.g., 
observation, rating scales) (Walker, et al.). Moreover, such students requiring intensive individualized 
behavioral supports may receive commercially available, scientifically-based approaches [e.g., 77?<? 
Incredible Years: Parents, Teachers, and Children Training Series (Webster-Stratton, 2001)] that must be 
delivered with fidelity to accomplish treatment goals. Thus, it may be concluded that the capacity of 
teachers to implement PBIS with fidelity appears to be important to positive social/emotional and 
academic outcomes of students requiring Tier III behavioral supports. We hypothesize that fidelity to 
PBIS delivered to students requiring intensive and individualized behavioral supports, particularly those 
receiving services under the special education category of emotional disturbance (ED), plays a 
statistically significant impact on the behavioral functioning of this population. Stated differently, we 
postulate that it will take fidelity of stmcture and process of PBIS to improve the responsiveness of 
students with ED and improve their social/emotional outcomes. 

Students with ED, by definition of antisocial behavior, require intensive (i.e., Tier 3) intervention 
efforts in their educational programming (Walker, et al., 1996). However, improving behavioral 
functioning outcomes for these students has proven difficult for a number of reasons. First, the number of 
students provided special education services under the category of ED has grown rapidly over the last 
decade with nearly 500,000 students identified as of 2002 (U.S. Department of Education, 2002a; 2002b). 
Unfortunately, parallel growth has not occurred in the number of highly qualified special education 
teachers prepared to meet the needs of students with ED (Katsiyannis, Zhang, & Conroy, 2003). This 
special education teacher shortage has led school districts to increase their reliance on hiring teachers with 
emergency licensures and those prepared through alternative certification routes to fill vacancies in 
special education classrooms (Katisyannis, Zhang, & Conroy). Although this practice occurs across 
special education categories, it occurs disproportionately for positions serving student populations with 
ED (Office of Special Education Programs 2002a; 2002b). The end result is that many teachers in ED 
classrooms may not be adequately prepared to carry out the demands of planning and providing intensive 
behavioral interventions with fidelity. Second, although students with ED share the characteristics of 
marked and long standing problem behaviors, the specific nature of their behavioral issues can vary 
widely (e.g., physical aggression, impulsivity, depression, anxiety). This range of emotional and 
behavioral problems requires professionals to be well versed across a wide variety of intensive 
instructional and behavioral approaches, and coordinate implementation of scientifically- and evidence- 
based behavioral approaches with fidelity. Even teachers meeting highly qualified standards of 
preparation for working with students with special educational needs may not possess this wide array and 
depth of key skills (U.S. Department of Education, 2002b; Office of Special Education Programs, 2002b). 
Third, over 50% of students with ED are educated in a separate, self-contained setting (Office of Special 
Education Rehabilitative Services, 2002). So although PBIS is designed to be a school-wide model, often 
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students in self-contained settings receive only intensive tertiary-level interventions (Hawken & O’Neill, 
2006). Taken together, these challenges place school administrators in the likely position of having to 
provide additional training and support to their teachers of students with ED. Thus, the role of 
professional development is of critical importance to build the capacity of special education teachers to 
meet the needs of students with emotional and behavioral disabilities. 

In the present study we utilized a pre-post naturalistic design to examine whether or not an 
intensive and sustained model of professional development in PBIS for teachers of students with ED 
would have a meaningful effect on the behavioral functioning of their students. We also explored 
whether teacher fidelity of PBIS implementation enhanced or constrained student changes in behavioral 
functioning (O’Donnell, 2008). In this context, the two purposes of the current study were to: 1) 
investigate the correlation of implementing PBIS and the behavioral functioning of students with ED 
served in self-contained settings; and 2) examine the extent to which teacher fidelity of PBIS 
implementation enhances or constrains student behavioral functioning. 

Method 


Participants 

Student participants. Thirty-seven public school students (29 males and 8 females) receiving 
special education services for ED in an urban, northwestern city participated in this study. The participant 
sample was drawn from self-contained classrooms serving students with ED across five different 
placements: one elementary school (n =7), two middle schools (n = 12), one high school (n = 4), and one 
separate day school for 1 st to 12 th grade students in = 14). In order to ensure that the teachers had a 
thorough knowledge of student strengths and problem behaviors, only students who had attended the self- 
contained cla ssrooms for at least two months were included in the study. No students were excluded from 
the study due to this reason, however. Ethnic breakdowns were 65% Caucasian (n = 24), 30% African- 
American (n = 11), and 5% Hispanic (n = 2). Ages of students ranged from 7 to 16 years, with a mean of 
13.2 (SO = 2.8). There were not statistically significant outcome differences across the five placements. 

Teacher participants. Eight teachers of participating students completed ratings of student social 
and emotional problem behaviors. The numbers of teacher participants at the elementary, middle, high, 
and separate day school were 1 (13%), 2 (25%), 1 (13%), and 4 (50%), respectively. At the beginning of 
the study, all schools of participating teachers had been implementing PBIS for at least two years and 
continued implementation over the year of the present study, but with varying degrees of fidelity. 

Teacher fidelity to intensive PBIS practices (detailed below) ranged from 44 to 100%, with an average of 
80% (SD = 19.9%). All of the teachers were female. The number of years teaching students with ED 
ranged from 1 to 19, with an average of 6.6 years (SD = 6.8). Teacher caseloads ranged from 6 to 20, 
with a mean of 11.5 (SD = 3.6) students. Caseload represents the number of students taught throughout 
the day, and does not necessarily equal class size. The number of participating students per teacher 
ranged from 2 to 7, with a mean of 4.6 (SD = 1.9). 

All participating teachers held special education teaching endorsements. However, given that 
preparation requirements to teach in special education settings in Washington State are non-categorical, 
teachers are considered qualified to work with students with ED when they hold an endorsement in the 
general area of special education. The majority of teacher training programs in Washington State prepare 
educators to work with students with learning disabilities, rather than those with ED. Although some 
teachers of these students have coursework specific to this population, none of the teachers in the present 
investigation held categorical certification in the area of ED. 
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Trainers. The two trainers in this study were university level researchers, and are the two lead 
authors of the present study, with expertise in positive behavioral interventions and supports and 
emotional disturbance. The trainers had a combined 45 years of expertise in PBIS and designing supports 
to meet the academic and behavioral needs of youth with ED. They had successfully implemented and 
sustained PBIS systems in many professional roles over their careers as special educators serving students 
with ED, technical assistance providers to schools and mental health agencies, university instmctors, and 
researchers. Given their expertise related to PBIS and ED, they received funding from the state of 
Washington Office of the Superintendent of Public Instmction (OSPI) to develop a model for building the 
capacity of teachers of students with ED to implement and sustain PBIS. They planned and conducted a 
training model in PBIS with the teacher participants over one school year. These trainers also served as 
observers and evaluators of teacher fidelity of PBIS implementation. 


Measures 

The Child Behavior Checklist: Teacher’s Report Form (TRF; Achenbach, 2001) was used to 
measure the behavioral functioning of participants. The TRF consists of 113 problem items, such as 
difficulty following directions, disturbing other pupils, and dismpting class discipline. The teacher rates 
the child on each item and indicates the severity of the problem on a three-point Likert-type scale ranging 
from 0 (Not True) to 2 (Very True or Often True). The TRF scoring profile provides a total scale score 
(Total Problems), two broad-band scale scores (Internalizing and Externalizing), and eight syndrome 
subscale scores (Withdrawn, Somatic Complaints, Anxious/Depressed, Social Problems, Thought 
Problems, Attention Problems, Rule-Breaking Behavior, and Aggressive Behavior). The broad-band 
Internalizing scale score is based on the sum of the Withdrawn, Somatic Complaints, and 
Anxious/Depressed scale scores. The broad-band Externalizing scale score is based on the Rule-Breaking 
Behavior and Aggressive Behavior scale scores. The Social Problems, Thought Problems, and Attention 
Problems syndrome subscale scores are not included on either the broad-band Internalizing or 
Externalizing scale scores. The TRF test-retest and internal consistency values for the broad and 
syndrome scales are reported in the test manual as ranging from .62 to .96 and .72 to .95, respectively 
(Achenbach). In the present study, internal consistency of the scales was satisfactory with Cronbach’s 
alpha noted as .76 for Internalizing, .84 for Externalizing, and .83 for Total Problems. Special education 
teachers serving students with ED in self-contained classrooms completed the TRF for each participating 
student in December 2005 and June of 2006. Teachers did not complete TRF protocols for students 
whom they had known for less than two months. A two-hour training session familiarized teachers with 
standardized assessment of student behavior and the structure (i.e., item formats) and specific instmctions 
for completing the measure. Teachers were given two weeks to complete the TRF. Each student was 
rated independently by their own teacher who had received the PBIS training, and interobserver 
agreement was not assessed. A research assistant entered the scores into the computerized scoring 
program for the TRF. This program requires verification of every item through a double entry system, 
resulting in 100% agreement between the protocol and the final data entry confirmation. 

A modified version of the Teacher Knowledge and Skills Sur\>ey (TKSS; Cheney, Walker, & 
Blum, 2004) was used to ascertain fidelity of implementation related to PBIS. The TKSS consisted of 25 
items with a 5 point response scale. The TKSS was created as a self-report form; however for this study 
it was modified to make it an observational form. Item 9, for example, originally said, “I know what 
functional behavioral assessments are and how they are used to develop behavior intervention plans for 
students,” and was changed to say: “Teacher knows what functional behavioral assessments are and how 
they are used to develop behavior intervention plans for students.” The TKSS targets five teacher actions 
critical to strong implementation of PBIS with students with ED: 1) Specialized Behavior Support 
Strategies, 2) Behavior Screening Methods, Behavior Support Services, and Evaluation, 3) School-wide 
Discipline Process, 4) Individualized Curriculum and Modifications Supporting Students, and 5) Positive 
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Classroom Environment. The first teacher skill (i.e., Specialized Behavior Support Strategies) focused on 
teacher skills related to students requiring intensive PBIS, including functional behavioral assessments, 
data-based decision making for behavior programs, de-escalation of behavior problems, outside services 
to link to families, and self-monitoring strategies. The second teacher skill (i.e., Behavior Screening 
Methods, Behavior Support Services, and Evaluation) measured knowledge of how to screen students for 
behavior problems, access pre-referral teams, counseling services and data-based decision-making. The 
third TKSS teacher skill area (i.e., School-wide Discipline Process) measures knowledge of the school¬ 
wide PBIS discipline policies, referral process, school-wide discipline team function, and goals and 
objectives. The fourth TKSS teacher skill area (i.e., Individualized Curriculum and Modifications 
Supporting Students) measures skills in modifying the curriculum, crisis support, collaboration with IEP 
teams, evaluation of individualized programs, and providing support for cultural or gender differences. 
The final TKSS teacher skill area (i.e., Positive Classroom Environment) measures skills necessary to 
provide a positive classroom environment (i.e., teaching and reinforcing expectations, social skills, 
prompts, and cues to re mi nd students of behavioral expectations). 

University trainers independently rated teacher level of mastery of knowledge and skill from one 
(none or little) to five (mastery of knowledge or skill) for each item. The trainer would then circle the 
appropriate number corresponding to teacher level of mastery with this item. Ratings were based on four 
classroom observations and notes from 20 training sessions with participating teachers over a one-year 
span. Independent TKSS ratings were conducted by the trainers in June 2006, after teachers had received 
all 40 hours of professional development and the trainers had conducted four observations of each 
teacher. The four observations were spaced throughout the time period between December 2005 and June 
2006. Pearson Product Moment Correlations were used to estimate inter-rater reliability of the TKSS 
rating scores between the two PBIS trainers. The inter-rater agreement coefficient for the overall TKSS 
score was .94. The inter-rater reliability coefficients for the “Specialized Behavior Support Strategies,” 
“Behavior Screening Methods, Behavior Support Services, and Evaluation,” “School-wide Discipline 
Process,” “Individualized Curriculum and Modifications Supporting Students,” and “Positive Classroom 
Environment” teacher skills were .94, .96, 90, .93, and .92, respectively. Although strong inter-rater 
agreement was found, item disagreements were resolved through discussion and sharing evidence for 
ratings. After discussion and evidence sharing on item disagreements, raters reached consensus on the 
most accurate teacher rating on the individual item. 

Procedures 

Training in Positive Behavioral Inter\>entions and Supports. The present investigation represents 
a one-year professional development training project to increase the capacity of special education teachers 
to implement PBIS in self-contained programs for students with ED. Development of the training content 
was guided by three key sources. First, a needs assessment survey was conducted in December 2005 with 
participating special education teachers who served students with EBD in self-contained settings. The 
needs assessment was only used to guide instruction and was not conducted as a post-test. The survey 
was comprised of 10 items critical to the design and implementation of positive behavioral supports. 
Teachers rated their level of prior training on each item by indicating whether they had “no training,” 
“some training,” “sufficient training,” or “extensive training.” Of the 8 participating teachers, all 
described their level of training as less than “sufficient” in at least one of 8 important areas. These areas 
included designing effective reinforcers and consequences, conducting functional behavioral assessments, 
creating behavior intervention plans, teaching replacement behaviors, connecting individualized 
education plans (IEP) to positive behavior intervention plans (BIP), designing aversive therapy plans, 
conducting manifestation of determination reviews, and progress monitoring. Second, passage of recent 
legislation (e.g., Individuals with Disabilities Education Improvement Act of 2004; Elementary and 
Secondary Education Act—No Child Left Behind legislation of 2002) underscores the need to implement 
positive behavioral interventions and supports, conduct effective functional behavioral assessments, 
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monitor progress, report progress to parents, and make adjustments to intervention to address any 
unexpected lack of progress. Third, the importance of building capacity for teachers of students with ED 
in the areas of PBIS, functional behavioral assessment, and progress monitoring has been highlighted in 
the professional literature (Katsiyannis, Landrum, Bullock, & Vinton, 1997; U.S. Department of 
Education, 2002b). 

All eight self-contained ED teachers received the 20 training sessions together. The scope and 
sequence for the 20 sessions is provided in Appendix A. Training began with positive classroom 
management practices including the teaching of behavioral expectations and replacement behaviors (see 
Appendix A). Following this, training topics included assessing functions of behavior, developing 
positive behavior intervention plans that were explicitly linked to the functions of behavior, identifying 
and implementing research-based practices in the area of PBIS, and designing data collection procedures 
to i nf orm instmctional decisions. Training began in December 2005 and was carried out over the 
remainder of the school year, ending in early June 2006. The training consisted of 20 two-hour sessions 
presented by two university faculty members with expertise in the field of emotional and behavioral 
disorders (discussed previously). Each training session was comprised of three components. The first 
component was in lecture format and was a review of previously covered concepts. The second 
component was in lecture format and was the introduction of the new material. The third component was 
guided practice format, which was conducted in groups in which the teachers practiced the skills they had 
just learned. The mastery of the material was measured through TKSS scores, which were conducted by 
the two trainers during four classroom visitations. Teachers were paid a district-negotiated rate of pay for 
extra work for the 40 hours of training. In addition, teachers received payment for up to eight hours of 
extra pay for time spent beyond the workday to complete measures requested by the university trainers. 


Data Analysis 

Non-parametric paired samples statistical analyses were used to analyze pre- and post-test TRF 
scores. Non-parametric tests were chosen due to the small sample size. Pearson product correlations 
were then utilized to determine any significant relationships between TRF change scores and TKSS 
scores. Finally, ANCOVAs were conducted to determine the influence of teacher knowledge on 
decreases of student problem behavior. 

Results 


Purpose 1 

Two analyses were conducted to examine the impact of PBIS on the behavioral functioning of 
students with ED served in self-contained settings (Purpose 1). First, we conducted a non-parametric 
paired samples statistical analysis, the Wilcoxon Signed Ranks Test on the pre- and post-test TRF Scores 
of participants. The Wilcoxon Signed Ranks Test is more appropriate than paired samples /-tests in cases 
of unequal or small sample sizes, non-normal distributions, and unequal variances (Siegel & Castellan, 
1988). Pre- and post-test TRF scores and Wilcoxon Signed Ranks Test Zscores are found in Table 1. 
Inspection of Table 1 reveals that statistically significant reductions on the TRF Externalizing Problems 
(p = .028) and Total Problems (p = .004) broadband scores were found from pre- to post-test. Further, 
statistically significant reductions were found on the Thought Problems (p = .034), Attention Problems (p 
= .003), and Aggressive Behavior (p = .014) syndrome scores from pre- to post-test. 

Second, analyses were conducted to determine the percentage of students with ED (TV = 37) who 
met normative criteria for clinically significant internalizing, externalizing, and total behavioral problems 
at pre- and post-test. The established Child Behavior Checklist: Teacher’s Report Form criterion used to 
determine clinically significant behavioral problems was students’ internalizing, externalizing, or total 
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problems t- scores at or above 63. The percentage of students with ED who met this criterion prior to the 
intervention was compared to the percentage meeting the criterion after the intervention. Pre- and post¬ 
test differences were then compared. Statistically significant differences (X 2 = 15.9, 1, N - 37, p < .01) 
were found in the percentages of students with clinically significant internalizing behavior problems at 
pre-test (n = 20, 54%) and post-test (n = 17, 46%). Similarly, statistically significant differences (X 2 = 
12.6, 1, N = 37, p < .01) were found in the percentages of students with clinically significant externalizing 
behavior problems at pre-test (n = 28, 76%) and post-test (n = 25, 68%). Finally, statistically significant 
differences (X 2 = 14.5, 1, N = 37, p < .01) were found in the percentages of students with clinically 
significant total behavior problems at pre-test ( n = 28, 76%) and post-test (n = 23, 62%). 

Table 1 Pre- and Post-test TRF Scores and Wilcoxon Signed Ranks Test Z scores 


Measure 

Pre 

Post 

Change 

Z 

Sig. 

TRF Syndrome Scores 

Anxious/Depressed 

64.2 (10.0) 

62.8 (10.2) 

-1.4 

-1.3 

.195 

Withdrawn 

62.1 (6.8) 

60.2 (8.0) 

-1.9 

-1.8 

.067 

Somatic Complaints 

57.4 (7.6) 

56.1 (7.8) 

-1.3 

-1.0 

.294 

Social Problems 

64.8 (8.8) 

63.1 (8.3) 

-1.7 

-1.2 

.237 

Thought Problems 

62.1 (10.9) 

59.3 (10.4) 

-2.8 

-2.1 

.034* 

Attention Problems 

62.5 (8.2) 

59.6 (7.9) 

-2.9 

-3.0 

.003** 

Rule -Breaking Behavior 

66.8 (10.8) 

64.7 (10.6) 

-2.1 

-1.6 

.112 

Aggressive Behavior 

69.7 (12.7) 

66.9(12.1) 

-2.8 

-2.5 

.014* 

TRF Broad-Band Scores 

Internalizing 

64.4 (7.8) 

62.3 (9.7) 

-2.1 

-1.9 

.062 

Externalizing 

68.4 (10.1) 

66.1 (10.3) 

-2.3 

-2.2 

.028* 

TRF Total Score 

Total Problems 

67.8 (8.7) 

64.4 (9.3) 

-3.4 

-2.9 

.004** 


Note. Standard deviations are in parentheses. 
* p < .05 and ** p < .01. 
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Purpose 2 

Two analyses were conducted to examine the extent to which teacher fidelity of PBIS 
implementation enhances or constrains student behavioral functioning (Purpose 2). First, analyses were 
conducted to examine the extent to which the five teacher knowledge domain scores as well as the teacher 
knowledge total score correlated with the change in student problem behavior. Change scores were 
calculated for the TRF Externalizing Problems, Internalizing Problems, and Total Problems by 
subtracting student pre-test from post-test scores. All 18 correlations were statistically significant (p < 

.05) and inverse in direction, as expected. The strength, or magnitude, of correlations was assessed using 
the scale developed by Cohen (1988). Correlations of .10 to .29, .30 to .49, and .50 and above were 
considered small, moderate, and large, respectively. As indicated in Table 2, four correlations (22%) were 
large and 14 (78%) moderate in magnitude. Large inverse correlations were found between overall TKSS 
and TRF Total Problems (r = -.52). Inspection of TKSS domains revealed that large inverse correlations 
were found between Behavior Screening Methods, Behavior Support Services and Evaluation (r = -.51) 
and School-wide Discipline Process (r = -.54) and TRF Total Problems. The magnitude of the 
relationships between all TKSS scores (overall and domain scores) and student changes in student 
Externalizing Problems were moderate, ranging from -.35 (School-wide Discipline Process) to -.42 
(Behavior Screening Methods, Behavior Support Services, and Evaluation). The relationship between the 
TKSS School-wide Discipline Process score and student TRF Internalizing Problems change was large in 
magnitude (r = -.51,/? < .01). The magnitude of all other relationships between all TKSS scores (overall 
and domain scores) and student changes in student Internalizing Problems were moderate, ranging from - 
.33 (Individualized Curriculum and Modifications Supporting Students) to -.51 (School-wide Discipline 
Process). The correlation results are detailed in Table 2. 

Table 2 Correlations of TKSS Teacher Knowledge and Skills Survey Scores with Student TRF 
Externalizing, Internalizing, & Total Problems Change Scores 

TRF Change Scores 


TKSS domain/Overall 

Externalizing 

Problems 

Internalizing 

Problems 

Total 

Problems 

Specialized Behavior Support Strategies 

-.40* 

-.35* 

. 49 ** 

Behavior Screening Methods, 

Behavior Support Services, and Evaluation 

-.42** 

-.40* 

-.53** 

School-wide Discipline Process 

-.35* 

-.51* 

-.54** 

Individualized Curriculum and 

Modifications Supporting Students 

-.41* 

-.33* 

. 49 ** 

Positive Classroom Environment 

-.39* 

-.34* 

-. 47 ** 


Note. * p < .05 and ** p < .01. 
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Second, analyses of covariance (ANCOVAs) were conducted for overall problem behavior to 
determine the influence of teacher knowledge on decreases of student problem behavior. Problem 
behavior was conceptualized as the TRF Externalizing Problems score and the TRF Internalizing 
Problems broad-band scores. The covariate measured in the ANCOVAs was the TKSS teacher 
knowledge total score. Although the problem behavior scores showed statistic ally significant decreases 
(p < .05) overall, results indicated statistically significant interaction effects of the overall TKSS teacher 
knowledge total score on changes in pre- to post-test problem behavior (F = 10.43, p = .003). Means and 
analyses of covariance interaction effects by TKSS Teacher Knowledge and Skills Survey domains and 
overall are presented in Table 3. 

Table 3 Means and Analyses of Covariance Interaction Effects by TKSS Teacher Knowledge and Five 
Teacher Knowledge and Skills Survey Domains and Overall 


Interaction Effect 
on Problem Behavior a 


TKSS Fidelity Domain/Overall 

M 

SD 

F 

Sig. 

Specialized Behavior Support Strategies 

31.8 

10.2 

9.31 

.004* ** 

Behavior Screening Methods, Behavior Support 
Services, and Evaluation 

29.9 

5.2 

11.05 

.002** 

Discipline Process 

16.3 

1.8 

9.22 

.005** 

Individualized Curriculum and 

Modifications Supporting Students 

18.8 

7.2 

9.54 

.004** 

Positive Classroom Environment 

17.5 

4.1 

8.37 

.007** 

Total 

114.1 

27.1 

10.43 

.003** 


Note. a Interaction of TKSS x time x problem behavior. The construct of ‘problem behavior’ included 
TRF Internalizing Problems and TRF Externalizing Problems /-scores. 

** p < .01. 
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Identical ANCOVAs and post-hoc analyses were conducted to determine the interaction effects of 
the five teacher knowledge domain scores on the change of student problem behavior over time. The 
covariates measured in the ANCOVAs were the respective teacher knowledge domain scores. The results 
revealed statistically significant interaction effects between changes in pre- to post-test student problem 
behavior for all five teacher knowledge domain scores. The five teacher knowledge domain scores of 
Specialized Behavior Support Strategies; Behavior Screening Methods, Behavior Support Services, and 
Evaluation; School-wide Discipline Process; Individualized Curriculum and Modifications Supporting 
Students, and Positive Classroom Environment all showed statistically significant interaction effects on 
student problem behavior (p < .01). The results of the ANCOVAs are detailed in Table 3. 

Discussion 

The two purposes of the current study were to: 1) investigate the impact of PBIS on the 
behavioral functioning of students with ED (N = 37) served in self-contained settings; and 2) examine the 
extent to which teacher fidelity of PBIS implementation enhances or constrains student behavioral 
functioning. Although an abundance of research has demonstrated PBIS as an effective school-wide 
model, there is a paucity of research on the effects of PBIS specifically on students in self-contained 
settings. Moreover, it remains unclear to what extent teacher capacity to implement PBIS with fidelity 
impacts student behavioral outcomes. We hypothesized that teacher fidelity to the structure and process 
of PBIS would reduce overall behavioral problems of students with ED served in self-contained settings. 

Two main findings warrant discussion. The first finding is it appears PBIS is correlated with the 
behavioral functioning of students with ED served in self-contained settings. Statistically significant 
reductions were found in the pre- and post-test scores of students on the TRF syndrome scores of Thought 
Problems, Attention Problems, and Aggression. Further, statistically significant reductions in the number 
of students who met the criteria for clinically significant internalizing behavior problems, externalizing 
behavior problems, and total behavior problems were found. These data indicate that PBIS appeared to 
play a significant role in improving the behavioral functioning of students with ED. Although the impact 
of PBIS on the behavioral functioning of students with ED in self-contained settings has not yet been 
experimentally examined, to date, these data add to the small evidence-base for PBIS among students 
requiring intensive, individualized behavioral supports (Turnbull et al., 2002; Nelson, Martella, & 
Marchand-Martella, 2002). 

The second finding that merits discussion is that teacher fidelity to the structure and process of 
PBIS played a significant role in reducing student problem behaviors. Indeed, results of ANCOVAs 
revealed a statistically significant interaction on problem behavior. As indicated in Table 3, all five 
teacher domains of the TKSS demonstrated statistically significant interaction effects on student problem 
behavior at the p < .01 level. Thus, it appears all five PBIS domains as measured by the TKSS were 
significantly important to improving the behavioral functioning of students with ED served in self- 
contained settings. When the magnitude of the relationship between teacher implementation of PBIS and 
student behavioral functioning was examined, a large inverse relationship (r = -.52) was found between 
teacher implementation of PBIS (overall TKKS score) and TRF Total Problems. Moderate to large 
inverse correlations were found between TRF Internalizing, Externalizing, and Total Problems and all of 
the five PBIS domains (see Table 2) measured by the TKSS. These data indicate that building the 
capacity of teachers of students with ED served in self-contained settings to implement PBIS with fidelity 
may play a large role in improving the responsiveness of this population to individualized and intensive 
behavioral interventions. 

There are several implications of these findings. First, it appears that PBIS positively correlates 
with the behavior of students with ED in self-contained settings. Over 50% of students with ED are 
educated in a separate, self-contained setting (Office of Special Education Rehabilitative Services, 2002), 
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and often receive only intensive tertiary-level interventions of PBIS (Hawken & O’Neill, 2006). 

Therefore, our findings underscore the importance of building the capacity of teachers of students with 
ED in all components of school-wide PBIS, including primary-level interventions (i.e., school-wide 
expectations and school-wide rewards), and secondary-level interventions (e.g., check-in, checkout 
system, positive adult attention) as were measured by the TKSS (Hawken & O’Neill). 

Second, PBIS professional development activities and coaching of teachers of students with ED 
in self-contained settings appears to correlate with student behavioral functioning. Study findings 
demonstrate that the professional development activities were able to successfully build capacity of 
teachers, which resulted in improved student behavior outcomes. These findings have implications for 
elementary, middle, and high schools serving students with ED in self-contained settings. First, support 
when implementing a new program must be continued over an extended period of time. The professional 
development organization of PBIS components involved 20 two-hour sessions, plus in-classroom follow¬ 
up. Further, support for teachers will continue over the next school year, as it has been recommended that 
to successfully build capacity in a given area teacher support should continue for 12 to 15 months 
(National Institute of Child Health and Human Development, 2000). Second, the current shortage of 
well-prepared, highly qualified special education teachers, particularly those working with students with 
ED, makes professional development activities crucial (U. S. Department of Education, 2002b). 

Ongoing, high-quality training targeted to areas insufficient in teacher preparation must be conducted 
regularly and considered a high priority in order to increase teacher quality and therefore, student success. 

This investigation was limited in several ways. The first and most debilitating limitation is the 
research design used to examine PBIS and related behavior outcomes, which was not experimental in 
nature. Researchers of future investigations should use higher quality designs including regression 
discontinuity, randomized experimental designs, or quasi-experimental investigations, which will allow 
causal inferences to be made. Second, the 37 students participating in this study were located in an urban, 
northwestern city. Therefore, the participants are not demographically representative of the general 
population and the generalizability of the findings is limited. Further, although the study encompassed 
students in elementary, middle, and high schools, the mean participant age of 13.2 years (SD = 2.8) 
emphasizes an overrepresentation of older students. Third, student behavior outcomes were measured 
solely in the form of teacher rating scales, specifically the TRF. Future studies should incorporate other 
measures of student behavior, including self-assessment, parent-assessment, and observation. Further, 
other common measures of PBS should also be collected, including office discipline referrals, 
suspensions, expulsions, etc. Fourth, we did not include schoolwide [e.g., Schoolwide Evaluation Tool, 
Version 2.1 (Sugai, Eewis-Palmer, Todd & Horner, 2005)] measures of PBIS implementation nor did we 
control for child factors (e.g., SES) that may have influenced treatment outcomes. It is probable, for 
example, that quality of schoolwide PBIS systems may have accounted for a significant amount of the 
variance in teacher fidelity to Tier III PBIS. 

Our findings have implications for the challenge of moving PBIS to practice. Cook, Landmm, 
Tankersley, and Kauffman (2003) highlight that approaches may be rendered ineffective or counter¬ 
productive if not used with adequate dosage (amount of treatment) or when implemented without 
adequate fidelity. Placing this concern in context of the present investigation, despite almost a full year of 
training on PBIS, teachers implementing with less skill (i.e., low fidelity) tended to not experience large 
improvements in student behavioral functioning commensurate with their colleagues implementing with 
high levels of skill (i.e., high fidelity). Thus, these teachers may feel justified in concluding that either 
PBIS does not work very wel or at all with students with ED served in self-contained settings. Limited 
or no consideration to fidelity of intervention is a large threat to internal validity; without consideration of 
level of fidelity it is difficult to ascertain whether the intervention was responsible for enhanced or 
constrained treatment outcomes. We draw upon and concur with the more than 30 year-old findings of 
Hall and Loucks (1977) who found that those implementing an innovation vary on adherence to the 
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structure and quality of implementation of the innovation. Our findings underscore that student 
behavioral functioning outcomes varied significantly according to how well PBIS was delivered. 

A related implication centers on the professional development needed to implement and sustain 
evidence-based approaches in schools. In a national study of university-based teacher education 
programs, Levine (2006) found that many students seem to be graduating from teacher education 
programs without the skills and knowledge they need to be effective teachers. He found that three out of 
five teacher education alumni surveyed (62%) indicated that schools of education do not prepare their 
graduates to cope with the realities of today’s classrooms, particularly in the area of behavior 
management. Moreover, only 30% of principals said that their teachers are very or moderately well 
prepared to address the needs of students with disabilities. Cook and colleagues (2003) state it well: 
“Given the many pressures, time constraints, and limited resources under which teachers typically work, 
it is little wonder that contemporary teacher preparation programs and training results in many effective 
techniques being quickly abandoned, not used, or used incorrectly” (p. 352). 

With the identification of specific gaps in teacher knowledge and skills regarding the provision of 
PBIS to students with ED, we were able to design training to target the specific needs of participating 
educators. The four features identified in the national study of special education personnel needs 
(discussed previously) provided the framework for training activities in this partnership (SPeNSE, 2002). 
These features guided this partnership in the following four ways. First, professional development was 
ongoing and included a broad range of specified topics. The needs assessment process used in the present 
partnership informed the design of the content of the training sessions so that multiple topics from areas 
identified by teachers of students with ED as needing further training were included. Second, training 
was immediately relevant and of high quality. Once areas of need were identified by the partnership 
participants, training was targeted to bring in evidenced based practices that met their current classroom 
challenges. Third, training time was allowed for planning and implementation. Time was allowed for the 
trainers to guide participants as they adapted session materials to best meet the specific needs of their 
unique classroom situations. Additionally, allotted time for planning and assistance ensured that those 
with the least amount of background training received the necessary assistance for correct implementation 
of the positive behavioral approaches. Fourth, opportunities for informal assistance were included. 
Visitations to classrooms provided an avenue for giving additional clarification regarding accuracy of 
implementation of training. Furthermore, informal observations were used to provide acknowledgement 
to teachers for a job well done. 

In summary, the implementation of PBIS positively correlated with the behavior of elementary, 
middle, and high school students with ED in self-contained settings. Teacher fidelity to PBIS played a 
large and statistically significant role in improving the behavioral functbning of students with ED. 
Building the capacity of teachers serving self-contained students with ED to implement and sustain PBIS 
should involve not only workshops, but ongoing coaching and attention to fidelity of implementation. 

We conclude that such efforts may lead to significant reductions in the behavior problems of students 
with ED. 
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Appendix A: Scope and Sequence of 20 PBIS Training Sessions 
Session 1: Getting started, overview of PBIS 

Session 2: Overview of PBIS, evaluating your own capacity to implement and sustain PBIS 
Session 3: Routines and Ecological Arrangements 
Session 4: Expectations 

Session 5: Developing a Plan for Responding to Problem Behavior 

Session 6: Social Skills Instruction/Teaching replacement behaviors (Strategic and Intensive) 
Session 7: Self-Management (Strategic and Intensive) 

Session 8: Linking FBAs to IEPs & BIPs 

Session 9: Using data to make decisions about PBS & BIPs, Helping others to make decisions 
Session 10: Aversive Therapy and Manifestation Determination 

Session 11: Pre-Functional Assessment Considerations, Making Sure Core Behavior Supports (e.g., 

routines, ecology, and expectations) are Strong 

Session 12: Functional Behavioral Assessment I 

Session 13: Functional Behavioral Assessment II 

Session 14: Behavior Intervention Plans I 

Session 15: Behavior Intervention Plans II 

Session 16: Progress Monitoring: Academic 

Session 17: Progress Monitoring: Behavioral 

Session 18: Determining Responsiveness to Intervention 

Session 19: Scientifically-Based Strategic and Intensive Behavioral Interventions 
Session 20: Meeting the academic needs of students with ED 
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